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functions  with  the  tactical  missions  and  warfighting  functions/tasks  that  the  system  supports. 
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the  mission  context  at  the  time  of  fielding.”  Further,  it  facilitates  the  assessment  of  system 
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Section  231  of  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2007 
directed  a  review,  and  amendment,  if 
necessary,  of  defense  acquisition  test  and 
evaluation  (T&E)  policies  and  practices. 
In  response  to  Section  231,  the  Office  of  the  Deputy 
Under  Secretary  of  Defense  for  Acquisition,  Technol¬ 
ogy  and  Logistics  (USD  [AT&L])  produced  a 
Department  of  Defense  (DoD)  Report  to  Congress 
on  Policies  and  Practices  for  Test  and  Evaluation, 
dated  July  17,  2007,  in  order  to  satisfy  this  legal 
mandate.  This  report  discussed  eight  key  T&E 
principles. 

The  intent  of  this  article  is  to  describe  how  the  U.S. 
Army  Test  and  Evaluation  Command  (ATEC)  is 
implementing  these  eight  principles.  The  principles 
along  with  ATEC  initiatives  for  implementing  the 
principles  are  provided  in  Table  1. 

The  first  of  these  principles  encourages  the  T&E 
community  to  broaden  its  focus  away  from  pass/fail, 
final  milestone  decision-oriented  assessments.  It  sug¬ 
gests  providing  periodic  assessments  to  the  materiel 
developer  with  emphasis  on  progress  made  versus 
requirements  met/not  met.  This  method  had  been 
successfully  applied  in  support  of  the  Ballistic  Missile 
Defense  System  and  Future  Combat  Systems  pro¬ 
grams.  A  similar  approach  is  used  for  Rapid  Acquisi¬ 
tion  Program  Systems,  where  capabilities  and  limita¬ 


tions  assessment  reports  are  updated  every  6  months 
with  highlights  of  the  ongoing  T&E  efforts. 

The  second  principle  attests  to  the  value  of  very 
early  test  events  to  obtain  initial  insights  into  the 
strengths  and  weaknesses  of  early  design  concepts.  It 
also  implies  that  more  emphasis  on  early  experimen¬ 
tal  design  for  these  test/experimentation  events 
should  result  in  testers  being  able  to  more  efficiently 
gather  information  on  system  performance.  The 
increased  anal)?tical  rigor  that  design  of  experiments 
brings  should  result  in  better  early  estimates  of 
system  performance  and  potential. 

The  ultimate  goal  of  integrated  testing  is  to  ensure 
that  all  stake  holders  (Program  Manager  [PM], 
developmental  testers,  operational  testers,  and  evalu¬ 
ators)  collaborate  so  that  all  can  use  the  data  from  any 
test  event  to  satisfy  their  needs.  There  are  complicat¬ 
ing  factors,  which  make  fully  attaining  this  goal 
difficult;  for  example,  developmental  testing  (DT)  is 
characterized  by  the  use  of  a  “test-fix-test”  process, 
which  allows  the  system  design  to  constantly  be 
improved  and  refined;  final  operational  testing  (OT) 
is  usually  conducted  with  systems  that  are  nearly  fully 
mature  and  are  “production  representative.”  However, 
we  have  made  great  strides  in  approaching  this  goal 
over  the  last  several  months.  Many  more  DT  events 
are  being  conducted  with  what  we  refer  to  as  more 
“operational  flavor.”  Soldiers  are  participating  in  many 
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Table  1.  How  to  achieve  the  Office  of  the  Secretary  of  Defense  policy  guidelines. 


Principle 

How  to  Implement 

Measure  improvements  to  mission  capability 
and  operational  support. 

Conduct  periodic  or  annual  assessments  of  observed 
levels  of  improvement.  Each  assessment  becomes 
the  “rolling  baseline”  for  the  last. 

Experiment  to  learn  strengths,  weaknesses,  and 
the  effect  on  operational  capabilities. 

Increased  emphasis  on  DT/OT  experimental  design: 
test  under  controlled  conditions  to  determine 
capabilities  and  limitations. 

Integrate  DT  and  OT. 

Consider  using  a  single  lest  event  to  address  both 
DT/OT  T&E  requirements.  Use  an  OT 
experimental  design  during  DT.  Give  DT  an 
operational  flavor  with  appropriate  threats  and 

Soldier  operators. 

Begin  early,  be  operationally  realistic,  and 
continue  tbrougbout  the  life  cycle. 

Get  involved  very  early  in  the  development  process. 
Liaison  with  the  PM  and  contractors.  Coordinate 
and  assist  the  combat  developer  in  developing 
requirements  and  rationale. 

Evaluate  in  the  mission  context  expected  at  the 
time  of  fielding. 

Ensure  an  operational  environment  with  realistic 
and  repre.sentative  threats  that  are  expected  at  the 
time  of  fielding  plus  5  years.  Apply  a  “mission- 
based  test  and  evaluation”  (MBT&E)  focus  to  the 
system  evaluation. 

Compare  to  current  mission  capabilities. 

Whenever  possible,  include  in  the  OT  experimental 
design  a  comparison  to  the  baseline  system.  Factors 
such  as  terrain,  mission,  operational  mode  summary 
and  mission  profile  (OMS/MP),  and  tactics, 
techniques,  and  procedures  (TTP)  must  be  held 
constant  between  the  current  system  and  the  system 
under  development.  A  model-test-model  approach 
may  provide  the  best  estimates  of  system 
comparison. 

Use  all  available  data  and  information. 

Develop  a  data  management  strategy  and  define  the 
data  early.  Use  any  credible  data  source:  historical 
data  as  well  as  data  from  contractor  testing.  DT, 
Mod/Sim,  etc.  for  system  evaluation.  Data  must  be 
properly  managed  and  documented  (using  a  data 
model)  to  facilitate  understanding  of  differing 
sources  of  like  data. 

Exploit  benefits  of  Modeling  and  Simulation 

Supplcmcnt/complcmcnl  live  testing  with  virtual 
and  constructive  simulations.  Plan  for  early 
verification,  validation,  and  accreditation  of  models 
and  resource  the  effort. 

DT  events  either  as  observer  subject  matter  experts 
(SMEs)  or  as  system  operators  if  safety  constraints 
allow. 

Early  involvement  of  the  T8cE  community  in  system 
development  has  many  potential  benefits.  This  allows 
the  Army  Evaluation  Center  (AEC),  for  example,  to 
provide  the  data  requirements  to  testers  early  on,  so 
they  can  structure  test  events  to  satisfy  OT  as  well  as 
DT  evaluation  requirements.  Some  benefits  that  we 
have  seen  include  the  following: 

•  contributions  in  drafting  the  T&E  portions  of  the 
system  specifications, 

•  technical  support  for  the  source  selection  process, 

•  assistance  in  the  development  of  more  realistic 
requirements, 

•  improved  contractor  understanding  of  require¬ 
ments, 

•  more  informed  and  balanced  assessment  reports. 


•  problems  identified  earlier  and  fixed  more  eco¬ 
nomically,  and 

•  a  much  more  timely  developmental  process. 

The  Stryker  program  is  an  excellent  example  of  the 
efficiencies  gained  through  early  involvement.  Hun¬ 
dreds  of  improvements  were  identified  and  fixed 
during  an  extensive,  early  T6cE  program  for  the  initial 
eight  Stryker  variants.  The  system  was  developed  and 
fielded  quickly  and  in  record  time. 

Evaluating  in  a  mission  context  has  many  benefits. 
In  the  past,  evaluation  products  have  frequently 
highlighted  technical  performance  requirements  that 
have  not  been  achieved.  The  senior  decision  maker  “so 
what”  question  that  sometimes  has  proven  difficult  to 
answer  is  “What  are  the  operational  mission  impacts  of 
not  meeting  these  requirements?”  To  facilitate  being 
able  to  address  this  question  and  others,  ATEC 
has  adopted  a  new  mission-based  T8cE  (MBT&E) 
methodology  for  all  its  programs. 
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We  do  not  always  have  the  resources  to  conduct  a 
live  test  in  which  the  system  under  test  is  compared 
with  the  baseline  system.  Also,  controlling  some 
factors  and  conditions  (level  of  unit  training,  different 
leadership  styles,  limited  sample  sizes,  learning  effects, 
etc.)  for  such  events  makes  them  difficult  to  execute 
and  analyze.  ATEC  has  adopted  a  “model-test-model” 
approach  as  a  preferred  evaluation  strategy  for  many  of 
these  types  of  system  tests. 

Traditionally,  ATEC  has  always  endeavored  to  use 
all  possible  data  sources  in  system  evaluation.  Utility  of 
some  of  these  data  has  been  improved  by  closer,  early 
coordination  between  the  contractor  and  the  T&E 
community.  Adoption  of  common  data  definitions  is  a 
new  initiative,  which  wiU  greatly  facilitate  the  utility  of 
this  data  exchange. 

The  utility  of  modeling  and  simulation  (Mod/Sim) 
in  support  of  the  T&E  process  has  expanded  markedly. 
Mod/Sim  has  been  used  effectively  as  a  “wrap-around” 
to  simulate  large,  joint  forces  for  major  tests.  Mod/Sim 
tools  have  extended  specialty  analysis  areas  such  as 
survivability;  reliability,  availability,  and  maintainabil¬ 
ity  (RAM);  and  integrated  logistic  support  (ILS). 
Force-on-force  models  have  been  used  to  augment  and 
extend  system  assessments.  This  is  a  large  growth  area 
for  T&E. 

Acceptance  of  these  principles  has  prompted  areas  of 
increased  emphasis  and  several  new  T&E  process 
innovations.  The  rest  of  this  article  focuses  on  four  of 
the  above-mentioned  initiatives:  MBT&E,  experimen¬ 
tal  design,  data  management  and  documentation,  and 
integrating  developmental  and  operational  testing. 
These  best  practices  are  essential  to  improving  our 
strategy  for  system  evaluation  and  to  enabling  the 
identification,  definition,  structuring,  and  prudent 
resourcing  of  the  areas  of  study  and  the  data  needed 
to  test  and  evaluate  an  Army  system.  ATEC  System 
Teams  (ASTs)  responsible  for  the  T&E  of  Army 
systems  address  these  initiatives  in  their  T&E  plans. 

MBT&E 

MBT&E  is  a  methodology  that  seeks  to  assess  the 
“strengths  and  weaknesses”  of  a  system  and  its 
components  and  “the  effect  on  operational  capabili¬ 
ties.”  Purposes  and  requirements  for  the  methodology 
were  derived  from  recent  T&E  policy-shaping  initia¬ 
tives  such  as  the  initiatives  shown  in  Table  1.  The 
overall  development  of  MBT&E  has  been  done  using 
an  interagency  working  group  that  assessed  individual 
mission-based  concepts,  developed  a  combined  meth¬ 
odology  from  these  concepts  and  from  T&E  policy 
requirements,  coordinated  the  methodology  with 
organizations  throughout  the  acquisition  community, 
executed  pilot  projects,  and  incorporated  comments 


and  lessons-learned  from  the  coordination  and  execu¬ 
tion  efforts.  The  result  is  a  baseline  MBT&E 
methodology  that  was  published  in  January  2009  and 
is  in  use  to  develop  ATEC  T&E  strategies  today. 
Continuing  methodology  development  is  leading  to 
improved  efficiencies  and  synchronization  of  the 
efforts  of  the  Services’  Operational  Test  Agencies 
(OTAs)  and  the  Director,  Operational  Test  and 
Evaluation  (DOT&E). 

MBT&E  consists  of  a  framework  and  a  procedure. 
The  MBT&E  framework  is  composed  of  four  main 
elements:  mission,  system,  evaluation,  and  test.  The 
methodology  links  the  information  in  these  elements 
to  create  a  T&E  strategy  that  associates  the  function¬ 
alities  a  system  must  possess  with  the  warfighting 
functionalities/capabilities  it  must  support.  Further,  it 
identifies  the  associated  T&E  measures  and  standards, 
operational  conditions,  and  the  data  required  to 
support  the  evaluation.  The  MBT&E  procedure  is  a 
step-by-step  description  of  activities  that  guide  the 
development  and  execution  of  the  T&E  strategy.  It 
describes  the  activities  for  the  following: 

•  analyzing  unit  missions  and  tasks, 

•  analyzing  the  materiel  system  of  systems  compo¬ 
nents  and  functionality, 

•  developing  T&E  measures  and  data  sources, 

•  executing  the  evaluation, 

•  and  reporting  of  unit  and  system  effectiveness, 
suitability  and  survivability. 

The  framework  and  procedure  provide  a  method  to 
conduct  the  following: 

•  planning  of  integrated  and  synergistic  T&E 
requirements, 

•  continuous  evaluation  from  program  inception 
through  deployment,  and 

•  evaluation  simultaneously  focused  on  system 
performance  and  warfighter  capabilities  in  the 
mission  context. 

Table  2  shows  an  example  of  the  MBT&E  frame¬ 
work  applied  to  an  unmanned  aerial  system  (UAS)  unit 
conducting  an  attack  by  fire  mission.  The  mission  is 
broken  down  into  three  “phases”:  (a)  conduct  recon¬ 
naissance,  (b)  direct  attack,  and  (c)  indirect  attack. 
Each  phase  is  further  broken  down  by  the  unit  tasks 
necessary  to  accomplish  the  mission.  The  mission, 
phases,  and  associated  unit  tasks  are  shown  in  the 
column  headings.  The  system  components  and  func¬ 
tions  are  shown  in  the  row  headings.  The  system 
functions  are  linked  to  the  tasks  by  the  addition  of  an 
evaluation  measure  in  the  intersecting  box  in  the  table. 
The  operational  context  of  the  task  is  applied  to  the 
system  functions  through  these  linkages  and  is  used  to 
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Table  2.  Mission-based  test  and  evaluation  (MBT&E)  framework  example — unmanned  aerial  system. 


UAS:  Unmanned  Aerial  System 

OM:  Operational  Measure 
'I’M:  I'cchnical  Measure 

VirSSION/TASKS 

Attack  by  Fire  (ART  7.5.1) 

Conduct  Reconnaissance 

Direct  Attack 

Indirect  Target 

Arrive  in  Area 
of  Operations 
(AO) 

Detect  &  Locate 
Surface  Targets 
(ART  3.2) 

Eitcl  Support 
10  Targeting 
(ART  2.4.1) 

Lethal  Direct 
Fire  Against 
Surface  Target 
t.\RT  2.4.1) 

Provide 

Fire 
Support 
(ART  3.3) 

2.3.2  Indirect 
Fire.  Surface- 
to-Surface 
(ART3.3.1.1) 

SYSTEM  COWONENT 

SYSTEM 

FUNCTION 

UAS 

Airframe 

Navigate  to 

AO 

-  Time  to 
Arrival  in  AO 

-  Navigation 
Accuracy 

Execute  search 
pattern 

-  Time  on 
Station 

Communication 

Equipment 

Send  call  for 
fires 

-  %  Lire 

Calls  Senty 
Received 

Sensor 

Detect  target 

-  %  Targets 

Detected 

-  Detection  Range 

Locate  target 

-  Location 

Accuracy 

Identify  target 

-  %  Targets 
Identified 

Laser  Designator 

Designate 

Taiget 

-  %  Targets  Hit 

-  Spot  .fitter 

Missile 

Destroy  Target 

-  %  Targets 

Killed 

M109 

Paladin 

Coninumication 

Equipment 

Receive  Firing 
Order 

-  Time  to 
Process 
l  ire  Order 

Idring  System 

Lire  Round 

-  %  I'iirgcts 

Killed 

develop  the  operational  conditions  addressed  in  the 
measure.  This  example  illustrates  the  design  of  a 
system  of  systems  evaluation  by  looking  at  the 
performance  of  the  UAS  to  conduct  a  direct  attack 
and  to  support  an  indirect  attack  using  a  M109 
Paladin.  It  also  illustrates  the  use  of  tasks  in  the  Army 
Universal  Task  List  to  provide  a  doctrinal  reference 
and  help  establish  measures  and  standards  used  in  the 
evaluation. 

The  achievement  of  an  integrated  MBT8cE  strategy 
comes  with  the  analysis  of  data  sources  to  support  the 
evaluation  of  the  measures.  Tal>/e  3  shows  the  measures 
from  Table  2  and  the  linkages  to  all  possible  data 
sources.  Possible  data  sources  include  contractor  tests, 
Mod/Sim,  hardware-in-the-loop,  and  interoperability 
certification.  Integrated  use  of  the  data  is  achieved  by 
identifying  the  right  type  of  data  at  the  right  time  in 
the  development  program.  Synergistic  use  of  the  data  is 
achieved  by  combining  the  data  from  more  than  one 
source  into  an  “accumulated”  knowledge  of  system  and 
task  performance. 


The  execution  of  the  MBTScE  pilot  projects  and 
ongoing  implementation  of  the  MBTScE  methodology 
within  ATEC  have  yielded  a  long  list  of  lessons  learned 
and  observations.  Lessons  learned  are  being  applied  to 
improve  the  development  and  robustness  of  MBT&E 
strategies.  Some  key  observations  include  the  following: 

•  Impact  of  system  performance  on  mission  task  is 
being  achieved. 

•  Task  context  is  critical  to  designing  integrated 
T&E  and  enabling  more  operationally  relevant 
developmental  test. 

•  Improved  efficiencies  and  aligning  of  expecta¬ 
tions  are  achieved  by  combat  and  materiel 
developer  participation  in  the  MBT&E  process. 

•  MBT&E  is  similar  to  the  processes  used  by  the 
U.S.  Navy’s  Operational  Test  and  Evaluation 
Force,  and  the  U.S.  Air  Force’s  Operational  Test 
and  Evaluation  Center. 

The  last  observation  has  led  to  an  effort  within  the 
Services’  OTAs  and  DOT&E  to  develop  a  common 
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Table  3.  MBT&E  framework  example — measure  to  data  source 
linkages. 


MEASURE 

DATA  SOURCE 

Contractor  Tests 

Modeling  & 

Simulation 

Interoperability 

Testing 

Hardware-In-The- 

Loop 

DT  Flight  Tests 

Operational  Tests 

UAS  Time  to  Arrive  in  Area  of 
Operations 

X 

X 

UAS  Time  on  Stotion 

X 

X 

UAS  %  Targets  Detected 

X 

X 

UAS  %  Targets  Identified 

X 

X 

UAS*  Targets  Hit 

X 

X 

X 

UAS  %  Ta^ete  Killed 

X 

M1Q9  Paladin  %  Targets  Killed 

X 

M109  Paladin  Time  to  Process  Fire 
Order 

X 

X 

UAS  Navigation  Accuracy 

X 

UAS  Sensor  Detection  Range 

X 

X 

UAS  Target  Location  Accuracy 

X 

%  Fire  Calls  Sent  and  Received 

X 

process  with  the  goals  of  synchronizing  processes, 
products,  and  training;  and  enhancing  Joint  Services 
operational  testing. 

Experimental  design 

A  statement  of  the  problem  facing  our  analysts  is 
essentially  this:  Given  ever  increasing  resource  con¬ 
straints,  assess  to  the  extent  possible  the  effectiveness 
of  an  Army  system  under  the  fuU  range  of  operational 
conditions  within  which  it  is  intended  to  operate.  In 
ATEC,  we  are  dealing  with  this  problem  by  placing 
greater  emphasis  on  formal  experimental  designs  versus 
reliance,  sometimes  on  less  structured  or  short  “free- 
play”  demonstrations.  Experimental  designs  permit  a 
rigorous  structuring  of  test  events  such  that  the  full 
range  of  operational  conditions  can  be  addressed  while 
conserving  resources.  Moreover,  the  use  of  experimen¬ 
tal  designs  to  structure  test  events  allows  the  evaluator 
to  assess  from  “improvement  to  mission  capability  and 
operational  support,”  to  “learn  strengths,  weaknesses, 
and  the  effect  on  operational  capabilities”;  and  to  make 
comparisons  “to  current  mission  capabilities.”  Tools 
and  procedures  used  to  structure  test  events  include 
factorial  designs,  fractional  factorial  designs,  sample 
sizing,  and  statistical  analysis. 

Experimental  designs  are  represented  during  early 
command  reviews  of  the  evaluation  strategy  using  a 
matrix  showing  how  the  factors  and  conditions  are 
combined.  An  important  step  in  getting  to  the  matrix 


Table  4.  Factors  and  conditions  table  template. 


Factor  (F) 

Control 

Conditions  (C) 

Fi 

Control  Type 

Cr 

...C, 

F2 

Control  Type 

Cl 

...Ck 

F3 

Control  Type 

Cl 

...Cl 

Fn 

Control  Type 

Cl 

...  C. 

is  a  factors  and  conditions  table.  In  this  table,  each 
factor,  the  conditions  the  factor  may  assume,  and  the 
method  of  controlling  the  factor  is  given. 

Factors  may  be  systematically  varied,  uncontrolled, 
“tactically”  varied,  or  held  constant.  Tactically  varied 
factors  are  changed  in  accordance  with  unit  tactics, 
techniques,  and  procedures  and  have  the  same  status  as 
uncontrolled  factors,  unless  incorporated  into  the  test 
design.^  An  experimental  design  requires  that  at  least 
one  factor  is  systematically  varied.  If  all  of  the  factors  in 
the  table  are  tactically  varied  (and  not  incorporated  into 
a  test  design)  or  are  uncontrolled,  it  may  be  very 
difficult  to  explain  why  a  system  performed  well  or 
performed  poorly. 

Analysts  prepare  a  factors  and  conditions  table  and  a 
matrix  early  in  the  planning  process.  A  factors  and 
conditions  table  template  is  given  in  Taile  4.  An 
example  is  provided  in  Table  5.  The  system  represented 
in  Table  5  is  the  Global  Position  System  (GPS) 
Guided  Fuze.  Four  factors  (weapon,  temperature, 
quadrant  angle,  and  munitions)  are  systematically 
varied,  two  factors  (range  and  mission-oriented 
protective  posture  [MOPP])  are  held  constant,  and 
one  factor  (weather)  is  uncontrolled.  Given  the 
controlled  factors,  an  experimenter  could  assess  the 
effects  of  the  varied  factors  on  delivery  accuracy.  The 
factors  and  conditions  given  in  Table  5  are  represented 
in  a  matrix  as  a  full  factorial  design  in  Table  6  and  as  a 
fractional  factorial  design  in  Table  7.  Fractional 
factorial  designs  can  enable  savings  in  the  total  number 
of  test  executions  without  giving  up  any  information 
about  the  effects  of  the  factors  and  their  first-order 
interactions.  As  shown  in  Table  7,  there  is  a  33  percent 
(12/36)  reduction  in  the  number  of  rounds  required  to 
assess  the  effects  of  the  four  systematically  varied 
factors  on  accuracy.  In  these  examples,  each  combina¬ 
tion  of  conditions  is  executed  five  times  to  help  ensure 
test  conditions  are  representative. 

The  use  of  experimental  designs  to  structure  test 
events  allows  the  evaluator  to  assess  the  effects  of  the 
full  range  of  operational  conditions  on  evaluation 
measures,  assess  improvements  to  mission  capability 
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Table  5.  Global  Position  System  (GPS)  Guided  Fuze  system  factors  and  conditions  tabie. 


Factor 

Control 

Conditions 

Weapon 

Systematically  Varied 

M109A6,  M777A2* 

Temperature 

Systematically  Varied 

Hot,  Ambient,  Cold 

Quadrant  Angle 

Systematically  Varied 

High,  Low 

Munitions 

Systematically  Varied 

M795,  M549,M107 

Range 

Held  Constant 

90%  of  maximum 

MOPP 

Held  Constant 

MOPPO 

Weather 

Uncontrolled 

*  The  M109A6  is  the  155rtim  self-propelled  howitzer  known  as  Paladin;  the  M777  is  a  towed 
lightweight  155mm  howitzer. 


and  operational  support,  make  comparisons  to  cur¬ 
rently  fielded  equipment,  and  conserve  limited  re¬ 
sources. 

Data  documentation 

AEC  must  be  able  to  communicate  tbeir  data 
requirements  to  the  tester;  design  and  create  data  sets 
that  can  be  used  to  analyze  the  data  received  from  the 
tester;  and  ultimately,  archive  that  data  that  can  be 
used  for  accrediting  models  and  simulations  and  future 
comparative  analysis.  Tbe  role  of  the  system  evaluator 
within  the  AST  is  critical.  They  must  ultimately  decide 
what  data  are  needed,  how  to  get  that  data,  and  what 
the  source  of  that  data  is. 

ASTs  must  also  be  able  to  use  data  from  all  sources. 
They  must  be  able  to  use  data  from  past  assessments  in 
order  to  measure  improvements  in  functionality  and 
operational  capabilities.  They  must  also  be  able  to  use 
contractor  data  and  DT  data  in  order  to  identify 


deficiencies  early  in  the  development  process.  The 
crucial  first  step  toward  accomplishing  these  goals  is  to 
properly  manage  the  data  requirements  identified 
during  the  requirements  analysis. 

Well-managed  data  are  visible,  accessible,  and 
understandable.  Data  are  made  visible  if  they  are 
properly  tagged  with  descriptors.  The  DoD  Discovery 
Metadata  Specifications  can  be  used  to  belp  accom¬ 
plish  making  data  visible.  Data  are  accessible  if  they 
are  held  in  a  common  area  and  can  be  searched  and 
retrieved  by  the  members  of  tbe  interested  commu¬ 
nity.  Data  are  made  understandable  through  precise, 
well-formed  definitions  documented  in  a  data  dictio¬ 
nary. 

Currently,  increased  emphasis  is  being  placed  on 
making  data  understandable.  Analysts  in  AEC  are 
asked  to  follow  naming  and  defining  conventions  when 
developing  data  dictionaries.^  ^  In  the  data  dictionar¬ 
ies,  the  analysts  name  and  define  the  objects  and  events 


Table  6.  Global  Position  System  (GPS)  Guided  Fuze  system  full  factorial  design. 


Temperature 

Weapon 

Type 

Quadrant  Angle  High 

Quadrant  Angle  Low 

M795 

M549 

M107 

M795 

M549 

M107 

Hot 

M109A6 

5 

5 

5 

5 

5 

5 

Hot 

M777A2 

5 

5 

5 

5 

5 

5 

Ambient 

M109A6 

5 

5 

5 

.-i 

5 

5 

Ambient 

M777A2 

5 

5 

5 

5 

5 

5 

Cold 

M109A6 

5 

5 

5 

5 

5 

5 

Cold 

M777A2 

5 

5 

5 

5 

5 

5 

Table  7.  Global  Position  System  (GPS)  Guided  Fuze  system  fractional  factorial  design. 


Temperature 

Weapon 

Type 

Quadrant  Angle  High 

Quadrant  Angle  Low 

M795 

M549 

M107 

M795 

M549 

M107 

Hot 

M109A6 

5 

5 

5 

5 

Hot 

M777A2 

5 

5 

5 

5 

Ambient 

M109A6 

5 

5 

5 

5 

Ambient 

M777A2 

5 

5 

5 

5 

Cold 

M109A6 

5 

5 

5 

5 

Cold 

M777A2 

5 

5 

5 

5 
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being  studied  (e.g.,  unit,  mission,  task,  sensor  type), 
and  name  and  define  the  data  elements  (e.g.,  unit 
identifier,  mission  date-time,  task  outcome  code, 
sensor  type  name)  used  to  describe  the  objects  and 
events  under  study. 

AEC  must  be  able  to  communicate  data  require¬ 
ments  to  the  tester,  use  the  data  received  from  the 
tester,  and  archive  data  for  future  use.  ASTs  must  also 
be  able  to  use  data  from  diverse  sources  including 
historical  data  and  contractor  data.  This  is  made 
possible  if  data  are  understandable  and  are  documented 
in  a  data  dictionary  that  is  shared  throughout  the  test 
and  combat  and  materiel  development  communities. 

Combined  and  integrated  DT  and  OT 

Combining  or  integrating  DT  and  OT  is  becoming 
more  and  more  necessary  in  order  to  address  the 
greatest  number  of  evaluation  questions  vrith  the  least 
use  of  limited  T6cE  resources.  Combined  DT  and  OT 
describe  a  single  test  event  that  produces  data  to 
answer  both  developmental  and  operational  system 
issues  and  is  usually  conducted  as  a  series  of  distinct 
DT  and  OT  phases.  Integrated  DT  and  OT  describe  a 
single-phased  event  that  generates  data  to  address 
developmental  and  operational  issues  simultaneously 
under  operational  conditions.  Integrated  DT  and  OT 
have  the  potential  to  answer  both  DT  and  OT  issues 
more  efficiently  in  terms  of  the  time  and  resources 
normally  required  by  separate  tests.  It  is  also  the  most 
difficult  type  of  testing  to  execute  as  it  requires 
maximum  coordination  and  cooperation  among  all 
members  of  the  test  community.  AEC  analysts  are 
required  to  consider  integrating  DT/OT  in  the 
evaluation  strategy  while  considering  the  following 
guidelines: 

•  FuUy  consider  aU  safety  issues. 

•  Achieve  maximum  cooperation  among  test 
team(s). 

•  Consider  use  of  highly  experienced,  soldier 
SMEs  for  early  test  events. 

•  Consider  use  of  typical  user  soldiers  for  later  test 
events. 

•  Enhance  operational  realism  of  DT  events, 
whenever  possible. 

•  Use  a  single  experimental  design  to  answer  both 
DT  and  OT  questions. 

Currently,  the  most  acceptable  form  of  integrating 
DT  and  OT  is  to  enhance  the  operational  realism  of 
the  DT.  This  is  usually  done  by  considering  realistic 
threats  and  observing  operational  conditions  and 
scenarios  specified  in  the  Operational  Mode  Sum¬ 
mary/Mission  Profile.  Sometimes  imparting  a  real¬ 
istic  operational  “flavor”  to  a  test  is  relatively  easy. 


One  example  would  be  the  varying  of  message  load 
when  testing  interoperability  between  two  commu¬ 
nication  systems.  Imparting  a  realistic  operational 
“flavor”  can  also  be  difficult.  For  example,  the 
required  use  of  soldier  operators  requires  a  safety 
release  that  may  not  be  available  during  DT. 

A  question  will  always  arise  as  to  whether  the 
conditions  under  which  testing  occurs  are  sufficiently 
operational  to  permit  the  use  of  data  generated  at  a  DT 
event  to  answer  operational  issues.  The  AST  will  make 
this  judgment.  If  a  single-phased  event  cannot  be  used 
to  generate  data  to  address  developmental  and 
operational  issues  simultaneously  under  operational 
conditions,  a  combined  event  with  distinct  DT  and 
OT  phases  can  be  implemented.  Conducting  DT  with 
deference  to  operational  realism  followed  by  an 
operational  phase  may  be  much  easier  to  execute. 

An  experimental  design  must  be  used  if  the  goal  of 
testing  is  to  predict  how  a  system  will  perform  under 
different  conditions.  An  experimental  design  that 
accommodates  the  needs  of  the  developmental  tester 
and  operational  tester  must  be  used  for  integrated 
testing.  In  contrast,  combined  DT/OT  can  employ 
separate  experimental  designs  for  the  different  DT  and 
OT  phases. 

Summary 

The  eight  OSD  guidelines  introduced  in  the  2007 
DoD  Report  to  Congress  on  Policies  and  Practices  for 
Test  and  Evaluation  provide  a  basis  to  better  fulfill  the 
key  objectives  of  Defense  acquisition:  to  acquire  quality 
products  that  satisfy  user  needs,  to  do  so  with 
measurable  improvements  to  mission  capability  and 
operational  support,  to  accomplish  this  in  a  timely 
manner,  and  to  ensure  purchase  at  a  fair  and  reasonable 
price.  AEC  has  undertaken  to  implement  these 
principles  by  introducing  tbe  initiatives  in  Table  1. 
Tbe  initiatives  include  MBTScE,  experimental  design, 
data  documentation,  and  integrating  DT  and  OT. 
These  best  practices  are  essential  to  our  evaluation 
strategy.  They  enable  the  identification,  definition, 
structuring,  and  efficient  resourcing  of  the  areas  of 
study  and  the  data  needed  to  test  and  evaluate  an  Army 
system  and  are  addressed  by  ASTs  early  in  the 
planning  process.  □ 
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Endnotes 

^The  stimulus  for  a  unit  to  tactically  vary  comes  from  some  external 
change  in  the  test  unit’s  physical  or  tactical  environment.  If  the  conditions 
resulting  in  a  tactical  change  are  made  part  of  the  test  design,  there  will  be 
start  and  finish  times  for  the  conditions  assumed  for  a  tactically  varied 
factor.  Data  collected  between  the  start  and  finish  time  for  a  specific 
condition  will  have  had  that  condition  held  constant. 

^International  Organization  for  Standardization  and  International 
Electrotechnical  Commission  (ISO/IEC)  international  standard  11179  - 
Metadata  registries  (MDR). 

^DOD  Directive  8320.02  Data  Sharing  in  a  Net-Centric  Department 
of  Defense,  April  23,  2007. 

'’DOD  8320. 02-G  Guidance  for  implementing  Net-Centric  Data 
Sharing,  April  12,  2006. 


368  ITEA  Journal 


